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LONG-TERM  GOALS 

Growing  concerns  that  aquatic  noise  produced  during  naval  exercises  and  offshore  seismic  surveys 
may  be  hannful  to  marine  mammals,  have  led  an  increasing  number  of  regulating  agencies  to  request 
mitigation  measures  when  issuing  pennits  for  such  surveys  in  their  nations’  EEZ.  The  most  common 
measure  is  to  implement  a  “marine  mammal  watch”,  a  team  of  observers  that  scans  the  ship’s  environs 
for  signs  of  presence  of  marine  mammals  to  trigger  a  shutdown  of  the  hydroacoustic  source  when 
marine  mammals  are  entering  a  predefined  exclusion  zone. 

Marine  mammal  observers  usually  scan  the  ship’s  environs  for  whales  using  binoculars  or  the  naked 
eye.  Sightings  mostly  rely  on  spotting  a  whale’s  blow,  which  might  rise  to  a  height  of  several  meters 
but  is  visible  for  a  few  seconds  only.  Hence,  in  combination  with  the  whales’  prolonged  dives,  sighting 
opportunities  are  rare,  which,  in  addition  to  the  limited  field  of  view  and  finite  attention  span  of  human 
observers,  renders  this  method  far  from  optimal,  even  during  fair  weather  and  daytime.  During 
darkness  it  is  not  feasible. 

Our  long-term  goal  is  to  overcome  these  difficulties  and  to  develop  a  reliable,  automatic  whale 
detection  system  for  the  full  range  of  oceanic  environmental  conditions  (wind,  sea  surface  temperature) 
and  species.  To  this  end,  we  developed  a  ship-based  thermal  imaging  system  for  automated  marine 
mammal  detection,  consisting  of  an  actively  stabilized,  spinning  IR  camera  and  an  algorithm  that 
detects  whale  blows  on  the  basis  of  their  thennal  signature.  So  far,  this  technology  has  been  tailored  to 
and  tested  under  cold  (SST  <  10°C)  water  conditions  only,  as  this  is  where  the  technology  was 
expected  to  perform  best.  To  adapt  the  technology  to  warmer  environments  and  to  test  its  performance 
there  is  the  specific  goal  of  this  proposal. 
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OBJECTIVES 


This  project  aims  at  a  cost-efficient  test  of  the  thennal  limits  of  the  abovementioned  IR  based 
automatic  whale  detection  technology  by  attempting  to  capture  whale  spouts  during  the  northward 
humpback  whale  migration,  which  occurs  annually  rather  close  to  shore  near  North  Stradbroke  Island, 
Queensland,  Australia1.  Based  on  the  collected  data,  detector/classifier  optimization  shall  be  perfonned 
and  ensuing  system  performance  be  tested  by  subsequent  comparisons  with  concurrent  visual 
observations. 

APPROACH 

By  obtaining  continuous  IR  video  footage  during  two1  successive  northward  humpback  whale 
migrations  and  collecting  concurrent  independent  (double  blind)  visual  observations  (modified  cue 
counting),  a  data  base  will  be  created  to: 

•  Detennine  the  perfonnance  of  the  imaging  device  in  warm  waters; 

•  Detennine  the  perfonnance  of  the  existing  automatic  detector  in  warm  waters; 

•  Develop  optimized  auto-detection  algorithms  for  the  wann  water  realm; 

•  Detennine  the  perfonnance  of  the  optimized  detector  in  warm  waters; 

•  Generate  additional  baseline  data  on  locomotive  behavior  of  migrating  humpbacks  (providing  the 
proposed  technology  proves  technically  feasible); 

•  Generate,  for  the  first  time,  data  on  night  time  humpback  migration;  (providing  the  proposed 
technology  proves  technically  feasible); 

At  AWI,  key  individuals  for  this  project  are  Daniel  Zitterbart,  overlooking  the  system  development, 
and  Elke  Burkhardt,  overlooking  the  concunent  sighting  data  acquisition.  The  field  activities  in 
Australia  were  supported  by  Mike  Noad,  University  of  Queensland,  who  has  organized  many  whale 
surveys  off  North  Stradbroke  Island,  the  field  activities  on  Kauai  are  supported  by  Aude  Pacini, 
researcher  at  the  Mammal  Research  Program,  Hawaii  Institute  of  Marine  Biology,  University  of 
Hawaii,  School  of  Ocean  and  Earth  Science  and  Technology. 

WORK  COMPLETED 

The  2014  field  season  was  conducted  on  North  Stradbroke  Island,  Queensland,  Australia  between 
06.06.2014  and  20.07.2014,  with  a  primary  data  collection  period  from  14.06.2014  until  14.07.2014. 
The  main  aim  for  the  2014  field  season  was  to  collect  IR  video  footage  of  blowing  whales,  whilst 
concurrently  recording  their  position  with  visual  observations  through  a  theodolite.  We  deployed  a 
team  of  2  (IR  technology)  plus  6  (visual  observers)  researches,  with  additional  local  support.  The  high 
abundance  of  blows/whales  also  allowed  allocating  6  days  of  expedition  time  to  acquire  a  first  data  set 
for  (double  blind)  visual-MMO/IR  comparisons.  Further  details  are  given  in  the  2014  annual  report. 


1  Please  note,  that  for  reasons  laid  out  in  the  main  text  below,  we  modified  the  original  plan  (as  described  here) 
of  two  consecutive  field  seasons  at  Stradbroke  Island  in  2014  and  2015  to  one  field  season  at  Stradbroke  Island 
in  2014  and  one  field  season  on  Kauai  in  2016. 
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In  FY  2015,  documentation  of  this  field  study  was  prepared  (expedition  report  available  on  request) 
and  thermographic  data  was  screened  for  whale  blows  on  the  basis  of  MMO  sightings.  This  resulted  in 
a  substantial  data  set  (see  results)  which  provides  the  basis  for  algorithm  testing  and  development. 

Performance  of  the  two  components  of  the  cold  water  detection  algorithm  (i.e.,  the  detector  and  the 
classifier)  was  explored  using  the  acquired  warm  water  thermographic  data.  Perfonnance  differed 
significantly  for  these  two  stages  (see  RESULTS).  The  findings  instigated  the  development  of  a  new 
detector  featuring  improved  computational  perfonnance  which  we  aim  to  have  implemented  for  the 
next  field  trials. 

Based  on  the  experiences  made  during  the  first  field  trial,  we  developed  a  tool  to  synchronize 
timestamps  from  the  various  observation  tools  for  the  second  field  trial.  We  also  developed  various 
scripts  and  parsers  to  log  environmental  data  from  the  weather  station  and  visibility  recorder. 

Preparation  of  the  expedition  report  included  a  detailed  reflection  on  the  informative  values  of  the 
various  metrics  and  protocols  used,  including  consideration  biases  and  study  designs.  This  led  to  a 
reconsideration  of  the  requirements  for  potential  field  sites.  For  the  Stradbroke  Island  site  in 
particular,  visual  observations  are  facilitated  by  the  whales  preferred  migration  directions,  allowing  the 
observers  to  capture  nearly  all  animals  by  focusing  their  attention  on  a  very  narrow  strip  of  water.  This 
renders  detection  performance  statistics  for  visual  observers  from  the  Stradbroke  site  unfit  to  represent 
the  (aimed  for)  open  ocean  conditions,  where  whales  might  turn  up  anywhere  relative  to  the  ship. 

Based  on  these  findings,  we  sought  and  evaluated  new  potential  field  sites  with  less  predictability  re 
whale  behavior,  organized  on-site  support,  and  made  necessary  arrangements  to  execute  the  next  field 
study. 

RESULTS 

The  primary  goal  of  this  field  season  was  to  acquire  sufficient  data  to  be  able  to  train  the  SVM 
(support  vector  machine)  of  the  automatic  whale  detection  algorithm  under  wann  water  conditions. 

This  goal  (the  collection  of  sufficient  data)  has  been  fully  achieved.  In  a  first  screening  of  the  IR 
imagery,  we  visually  identified  2330  cues  (mostly  blows)  in  a  total  of  5 1 1  encounters  (pods). 

Particularly  with  regard  to  the  latter  visual-MMO/IR  comparisons,  it  became  clear  during  the  field 
period  and  the  ensuing  discussions  in  writing  the  field  report,  that  the  high  directionality  and 
abundance  of  whales  at  the  Australian  filed  side  is  in  fact  biasing  comparisons  between  MMOs  and  IR 
and  results  are  only  of  limited  power  when  trying  to  describe  relative  sighting  probabilities  for  an  open 
ocean  situation.  Hence  it  was  decided  to  abandon  the  Stradbroke  Island  field  site  and  to  seek  a 
location  where  whale  swim  directions  are  less  directional  and  animals  are  less  frequent,  yet  still 
reliably  present  close  to  the  coast.  After  evaluation  of  infonnation  on  the  Azores,  the  California  coast 
and  the  Hawaiian  Islands,  our  final  choice  narrowed  on  Kauai,  with  known  seasonal  presence  of 
humpbacks.  We  identified  field  sites  at  the  North  and  South  shore,  which  will  be  occupied  in  Jan-Mar 
2016. 

Applying  the  cold-water  tested  detection  algorithm  on  the  wann-water  data  in  Australia,  revealed  a 
lack  in  computational  performance.  The  simple  short-term-average/long-term-average  algorithm  that 
was  used  to  select  thennal  anomalies  which  might  resemble  a  whale  blow  responded  primarily  to 
thermal  noise  caused  by  breaking  waves  (which  were  very  frequent  due  to  surf  along  the  shore).  This 
resulted  in  thennal  anomalies  from  whales  being  ranked  not  high  enough,  to  pass  the  detection  step 
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and  being  forwarded  to  the  classifier.  Contrastingly,  the  classification  algorithm,  as  based  on  the  cold- 
water  trained  support-vector  machine,  was  performing  well  on  the  whale  blows  that  passed  the 
detection  step. 

To  overcome  the  detector  bottleneck,  we  developed  a  new  detection  algorithm  (publication  in 
preparation)  which  however  depends  on  increased  computational  power,  prohibiting  its 
implementation  in  Matlab™  but  requiring  the  implementations  of  core  components  in  C++. 

To  re-train  the  classification  algorithm,  we  reviewed  3053  thennal  anomalies  considered  by  MMOs  as 
whale  signatures  and  categorized  them  in  9  classes  (wave,  blow,  back,  breach,  slap,  boat,  undefined 
cue,  undecided,  deleted). 

We  tested  the  classification  perfonnance  using  a  deep-learning  convolutional  neural  net  (CNN)  to 
classify  20000  manually  annotated  thennal  anomalies  from  the  cold-water  regime  into  6  different 
classes  (wave,  blow,  structure,  bird,  ice  and  unknown).  It  turned  out,  that  the  results  of  the  SVM  and 
the  Neural  Network  were  comparable  in  terms  of  positively  detected  whale  blows  and  false  positives, 
yet  with  a  much  higher  computational  load  for  the  Neural  Network  than  the  for  the  SVM.  To  facilitate 
this  comparison,  a  framework  that  allows  to  train  a  CNN  and  classify  the  small  video  snippets  that  are 
produced  by  the  detector  using  this  CNN  was  developed.  This  data  will  be  used  in  addition  to  the 
warm-water  data  used  to  develop  a  classifier.  The  current  classifier  used  is  a  binary  decision  making 
support-vector-machine  (SVM). 

To  facilitate  easy  review  of  thermal  anomalies  classified  as  whale  blow  without  the  need  of  a 
workstation,  we  designed  and  developed  an  App  for  mobile  devices  (i.e.  iPad)  that  alerts  the  MMO  of 
a  suspected  whale  detection  and  presents  him  the  data  in  a  continuous  loop  including  a  geo-referenced 
display  of  the  detection  of  the  last  30  min. 

IMPACT/APPLICATIONS 

Our  deliberations  result  in  an  aspect  of  wider  relevance:  Performance  comparisons  across  different 
observation/  detection  platforms  (e.g.  MMOs,  AIMMMS  or  PAM)  require  an  a-priori  definition  of  a 
metric  bearing  relevant  informative  value.  This  metric  needs  careful  consideration  prior  to  going  in  the 
field,  taking  into  consideration  the  various  platform  specific  uncertainties  regarding  positioning.  Our 
results  suggest,  that  this  metric  should  mimic  the  operational  challenge  as  close  as  possible,  i.e. 
including  specific  exclusion  radii.  For  example,  an  “in-time  detection”  metric,  answering  YES  or  NO 
to  the  question  if  an  animal  was  detected  by  the  method  prior  to  entering  a  presumed  exclusion  zone, 
might  be  fit  to  quantify  performance  across  platform,  whereas  a  metric  comparing  PAM  and  visual 
detections  directly  cue  by  cue  will  fail,  as  these  are  likely  not  to  occur  simultaneously.  For  the  second 
field  season,  we  plan  implement  a  “mitigation  protocol”  and  corresponding  metric,  i.e.  MMO  and 
AIMMMS  operators  will  make  in-situ  mitigation  decision,  to  latter  compare  these  against  each  other. 
With  assistance  of  recorded  data,  sightings  can  be  verified  in  the  aftennath,  to  also  extract  the  amount 
of  false  alerts  for  each  method. 

TRANSITIONS 

Rheinmetall  Defence  Electronics,  Bremen,  Germany,  has  acquired  a  license  of  the  AWI-developed 
data  acquisition  software  and  current  cold-water  detection  kernel  and  is  marketing  the  entire  system 
(the  FIRST-NAVY  sensor  head  and  the  AWI-built  software  components)  under  the  acronym 
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AIMMMS  (Automatic  Infrared  Marine  Mammal  Mitigation  System).  Any  further  software 
developments  made  during  or  sponsored  by  ONR  through  ETAW  do  not  fall  under  the  current 
licensing  agreement. 

RELATED  PROJECTS 

We  have  meanwhile  been  funded  by  the  Gennan  Ministry  for  Education  and  Research  (BMBF)  to  test 
the  capability  of  a  thermographic  dual-band  IR  camera,  which  holds  the  potential  of  providing  an 
improved  S/N  ratio. 

PUBLICATIONS 

Note:  The  papers/reports  listed  hereinafter  describe  the  current  state  of  development  with  regard  to 
“cool  water”  deployments  and  basically  describes  the  prior  knowledge  that  is  being  entered  into  the 
ETAW  (i.e.  this)  project  by  AWI. 

Zitterbart  DP,  Kindermann  L,  Burkhardt  E,  Boebel  O  (2013)  Automatic  Round-the-Clock  Detection  of 
Whales  for  Mitigation  from  Underwater  Noise  Impacts.  PLoS  ONE  8(8):  e71217. 
doi:10.1371/journal.pone.0071217  [published,  refereed] 

Boebel  O.,  Bombosch  A.,  Burkhardt  E.,  Cammereri  A.,  Kindennann  L.,  Richter  S.,  Zitterbart  DP. 
MAPS:  Marine  Mammal  Perimeter  Surveillance.  In:  Boebel  O.,  editor.  The  Expedition  of  the 
Research  Vessel  "Polarstem"  to  the  Antarctic  in  2012/2013  (ANT-XXIX/2).  Bremerhaven: 
Reports  on  Polar  and  Marine  Research,  [in  press] 

Burkhardt  E.,  Lanfredi  L.,  Richter  S.,  Boebel  O.,  Kindermann  L.,  Zitterbart  DP.  MAPS:  Marine 

Mammal  Perimeter  Surveillance.  In:  Lucassen  M.,  editor.  The  Expedition  of  the  Research  Vessel 
"Polarstern"  to  the  Antarctic  in  2012  (ANT-XXVIII/4).  Bremerhaven:  Reports  on  Polar  and 
Marine  Research.  652 , 89  p.  2012  . 

Richter  S,  Muller  M,  Boebel  O,  Kindermann  L,  Zitterbart  DP  (2012)  MAPS:  Marine  Mammal 

Perimeter  Surveillance.  In:  Kattner  G,  editor.  The  Expedition  of  the  Research  Vessel  "Polarstern" 
to  the  Antarctic  in  2011/12  (ANT-XXVIII/2).  Bremerhaven:  Reports  on  Polar  and  Marine 
Research,  pp.  56. 

Boebel  O.,  Bombosch  A.,  Kindermann  L.,  Zitterbart  DP.  MAPS:  Marine  Mammal  Perimeter 

Surveillance  In:  Fahrbach  E,  editor.  The  Expedition  of  the  Research  Vessel  "Polarstem"  to  the 
Antarctic  in  2010/2011  (ANT-XXVII/2).  Bremerhaven:  Reports  on  Polar  and  Marine  Research, 
634,102  -  110,  2011.  (PDF) 

PATENTS 

Note:  The  patent  below  describes  the  current  state  of  technology  and  basically  describes  the  prior 
knowledge  that  is  being  entered  into  the  ETAW  (i.e.  this)  project  by  AWI. 

Zitterbart  DP,  Kindermann  L,  Boebel  O;  Method  for  automatic  real-time  monitoring  of  marine 
mammals,  US  Patent  8,941,729  B2  from  Jan  27  2015. 

Russian  and 
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HONORS/AWARDS/PRIZES 

Zitterbart  DP,  AWI,  „BRIESE-award  for  Marine  Science”.  Briese  Schiffahrts  GmbH  &  Co.  KG. 
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